Scaling law

Scaling law

In AI research, there has been several scaling laws regarding the compute, model’s capacity, etc.

Xiong2024temporal studies ‘temporal scaling law’, or “how the test loss of an LLM evolves as the training steps scale up.”

Song2024resource proposes a “resource model” that can potentially explain the scaling law.

Busbridge2025distillation studies the scaling law in Model distillation.